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March 6, 2014

MEMORANDUM FOR:                    Dr. Kathryn D. Sullivan
                                   Acting Under Secretary of Commerce for Oceans
                                      and Atmosphere           AJ/J;J '\.--        ~

FROM:                              Allen Crawley             ~
                                   Assistant Inspector General for Systems Acquisition
                                    and IT Security

SUBJECT:                           Audit of NOAA's Geostationary Operational Environmental
                                    Satellite-R Series Core Ground System Observations

As you are aware, we began conducting our audit of NOAA's Geostationary Operational
Environmental Satellite-R (GOES-R) Series program in June 2013. Our audit objective is to
assess the adequacy of GOES-R development activities, as the program completes the ground
system, the fabrication of flight instruments and the spacecraft, and transitions to system
integration and test. We are also monitoring NOAA's progress in developing and vetting with
stakeholders a comprehensive set of trade-off approaches to mitigate launch delays, as well as
its oversight of GOES-R systems engineering. This memorandum provides you with our initial
observations concerning the core ground system 1 that warrant your immediate attention.

The original core ground system development plan-initially expected to meet an October
2015 launch readiness date2 but later found to be significantly behind schedule-was replaced in
20 I I with the first re-plan. In June 2013, the first re-plan was also found to be significantly
behind schedule and has since been replaced with a second re-plan. In evaluating ground system
development, we found that the contractor, Harris Corporation, had ( I) made poor planning
assumptions for the first re-plan, (2) been unable to execute the first re-plan, and (3) been
inadequately transparent about its progress. Further, we found that NOAA oversight and
GOES-R program management did not sufficiently address problems with the first re-plan that
could lead to increased costs, and now NOAA may have to launch a satellite without all of the
core ground system capabilities implemented.




'The core ground system is part of the GOES-R ground segment project, which also includes an antenna system.
2
  Since 2009, October 2015 had been the latest date the satellite would be available for launch. On December 11,
2013, NOAA approved changing the latest date the satellite would be available for launch to the second quarter of
FY 2016. The launch readiness date remains October 20 15 but is now the earliest the satellite will be available for
launch. The life-cycle cost also increased, by $150 million, to slightly more than $ I I billion.
Our 2013 GOES-R audit report 3 identified a number of issues with the original ground system
development plan—the projection that the system would be completed 15 months after the
launch date, the risks introduced by the prolonged time it took to establish the first re-plan, and
the need for effective technical leadership and oversight. The first re-plan, adding $89 million to
the cost of the development effort, was approved by the GOES-R program office, validated by
independent oversight, and intended to put system development back on track for an October
2015 launch. However, 2 years later, the development of the ground system under the first re-
plan was projected to be completed 11 months after the launch date. As a result, we remain
concerned about the effectiveness of management and oversight of the twice re-planned
development effort.

According to the GOES-R managers, there are no-cost changes to implement the core ground
system’s second re-plan; however, we believe there may be cost overruns. 4 The delays in the
development of the core ground system have raised the possibility that only a partial set of its
capabilities will be available at launch, which would introduce new post-launch system
development and operational risks. These delays may also increase pressure for the program
office to take testing shortcuts to meet the launch date. In a worst case scenario, the program
office has a contingency to launch GOES-R without a full set of mission management
functions 5—with product delivery that does not meet users’ timeliness requirements and only
the primary ground station in operation (i.e., without the remote backup facility for contingency
operations.)

Based on previous performance, we believe that, without leadership’s attention, the core
ground system may not meet minimum requirements for launch in October 2015; at best, the
ground segment project will have to enact a substantial number of development contingencies
that would increase program risk. Given the small schedule gains achieved by the first re-plan, it
is unlikely that the second re-plan can achieve twice the pace of improvement of the first re-
plan, which is needed to complete the core ground system 6 by the launch readiness date. 7 We
estimate the earliest the core ground system will be completed is March 2016, 5 months after
the launch readiness date (see figure 1, next page). We therefore believe that you should
establish periodic discussions with both Departmental and contractor leadership to ensure the
core ground system will meet the October 2015 launch readiness date.

3
 U. S. Department of Commerce Office of Inspector General, April 25, 2013. Audit of Geostationary Operational
Environmental Sate/lite-R Series: Comprehensive Mitigation Approaches, Strong Systems Engineering, and Cost Controls Are
Needed to Reduce Risks of Coverage Gaps, OIG-13-024-A. Washington, DC: DOC OIG.
4
  Cost overruns occur when contractors’ incurred and planned costs exceed the expected costs of their contracts.
5
  Mission management functions are used by operators to control spacecraft navigation and instrument operation,
schedule daily and long term tasks, and monitor health and safety information as well as space-to-ground
communications.
6
  The capabilities to be completed by October 2015 are specified in the core ground system schedule baseline
finalized on September 1, 2013.
7
  The first re-plan needed to achieved about 30 percent improvement to regain 15 months of schedule to meet the
launch readiness date (15 months gain / 48 months remaining in schedule as of October 2011). However, the pace
of improvement by June 2013 was only 20 percent (4 months actually gained / 20 months elapsed since first re-plan
started). Double that pace of improvement, or 40 percent (11 months gain / 28 months remaining in schedule as of
June 2013), would be needed from the second re-plan to regain 11 months of schedule to meet the October 2015
launch date.


                                                            2
                Figure1. OIG Estimate of the Core Ground System Completion Date,
                Based on Progress Made in Implementing Core Ground System Re-Plan
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                                                               First Re-Plan                      • Cost: $89 million
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                                     Schedule summit                                 5     Second Re-plan
                                       (June 2013)            Second Re-Plan      months   • Potential cost overrun
                                                                                   late    • Potential reduced ground system capability


                                                                                            OIG Estimatea
                                                                                            In October 2015, core ground system is
                                                                                            projected to be 5 months behind schedule

Source: OIG analysis of NOAA data
a
  OIG’s estimate of months remaining to develop the core ground system on October 2015 assumes that (1) the
contractor was able to start the second re-plan immediately when it was determined that development was 11
months behind schedule (i.e., at a June 2013 schedule summit meeting) and (2) development time can be regained
at the same rate as was regained by the first re-plan (i.e., 4 months regained over 20 months duration).

A summary of our audit observations about the core ground system follows:

     The first re-plan did not give high priority to the development of functions essential
     for launch. The focus of the first re-plan was flight and ground segment
     interdependencies—specifically, to minimize the time ground system developers would have
     to wait for deliverables from the flight segment to complete their work, as this was the
     main reason that development was projected to miss the launch date by 15 months. The
     first re-plan should have prioritized development of functions that are essential for launch
     and production of the mission’s two key data products (i.e., cloud and moisture imagery)
     with the core ground system projected to be on the critical path 8 for 3 years through the
     operational readiness review. 9 This re-plan also should have included contingencies, which
     the second re-plan now has, in case a full complement of functions would not be ready for
     launch.

     The first re-plan was based on the contractor’s over-optimistic assumptions. When
     the ground segment project engineers measured how long it took the contractor to
     integrate and test the first mission management release and migrate it to the NOAA
     Satellite Operations Facility in June 2013, the measurement indicated that the contractor’s

8
  The critical path is the sequential series of tasks with the longest duration through the project end date. Any task
in the critical path that slips off-schedule will cause the project deadline to slip.
9
  The operational readiness review ensures flight and ground systems, operation personnel, and procedures are ready
for launch and operations.
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     estimates of task durations were unrealistic. In addition, because the first re-plan coupled
     mission management functions with development of the other core ground functions (e.g.,
     enterprise management, as well as data product generation and distribution 10), there were
     too few contractor mission management experts available for assignment to the multiple
     project development teams. Also, the contractor did not plan for enough computer
     hardware for its facility, limiting its ability to quickly isolate and resolve technical problems
     found in system releases being tested at the government’s ground station sites.

     The GOES-R program office waited too long to intercede to correct schedule slips
     with the first re-plan. The ground segment project management and engineers started
     closely monitoring contractor performance in June 2012, after the core ground system’s
     critical design review. Although reported earned value metrics 11 did not detect schedule
     performance problems, the project manager and engineers were becoming increasingly
     concerned that the contractor was exceeding staff allocations to complete milestones while
     deferring other work. Finally, in June 2013, the GOES-R program office and contractor held
     a schedule summit meeting that resulted in the second re-plan of core ground system
     development. However, earlier intervention was possible and could have lessened some of
     the adverse impacts such as potential additional cost and risks associated with launching
     without a completed core ground system. On a positive note, the program has recently
     established more intensive weekly and monthly monitoring processes in order to respond
     more quickly to schedule slips.

     Technical problems, multiple development plans, and inadequate contingency
     funding caused schedule slips. Program officials indicated that funds available for the
     ground segment project had decreased in FY 2013 due to sequestration and transfers to the
     flight project ($21 million and $25 million respectively). Also, technical issues—such as
     frequent fluctuations in data product requirements and inadequate commercial off-the-shelf
     product performance—coupled with the second re-plan resulted in schedule slips, because
     there were insufficient contingency funds to assemble additional staff when problems arose.

     The second re-plan focuses on essential functions, but time constraints could lead to
     delayed satellite operations. To allow the contractor to focus resources on functions
     essential for launch, the second re-plan decouples development of mission management
     functions from other core ground functions. The second re-plan also lessens the burden on
     mission management experts by not assigning them to work on multiple development
     teams. Because functions essential for launch will be built first, the program recently
     developed schedule risk contingencies or “off-ramps” that would move development of
     nonessential functions until after launch. However, delaying development of functions until
     after launch could delay the satellite from being fully operational when it is needed for
     weather observations.


10
   Enterprise management functions are responsible for the supervision, status reporting, common support services,
and infrastructure for the entire ground segment. Products generation functions receive raw data, metadata, and
ancillary data and generate data products and associated metadata; monitors product quality; and provides storage
for GOES-R products. Product distributions functions receive data products and routes them to various destinations.
11
   An earned value metric is an objective measurement of how much work has been accomplished on a project.
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We appreciate the cooperation of your staff during our review and look forward to receiving
additional information on the progress of the second re-plan. If you have any questions or
would like to discuss our observations in this memorandum, please do not hesitate to contact
me at (202) 482-1855 or Fred Meny at (202) 482-1931.

cc:    Patrick D. Gallagher Acting Deputy Secretary of Commerce
       Ellen Herbst, Chief Financial Officer and Assistant Secretary for Administration
       VADM Michael S. Devany, Deputy Under Secretary for Operations, NOAA
       Mary E. Kicza, Assistant Administrator for Satellite and Information Service, NOAA
       Greg Mandt, GOES-R System Program Director, NOAA
       Mack Cato, Director, Office of Audit and Information Management, NOAA
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